Serverless Application using AWS Lambda, API Gateway and DynamoDB and NodeJS

This is small sample project that demonstrates a Serverless Application. It uses the following AWS Services:

API Gateway
Lambda
DynamoDB

The repository of this project can be found on github here:

The application is used as a personal goal tracker. It stores daily attempts (with timestamps) and an overall score for the day. On DynamoDB the attempt data is stored on a table and another Materialized View table tracks the overall daily score and counts of each daily attempt.

Setting up AWS Lambda and API Gateway

In AWS we will be creating a Lambda Function, an API Gateway endpoint and 2 DynamoDB tables. To start we will need to create a Role in IAM that the Lambda function will use in order to communicate with DynamoDB and CloudWatch. In IAM – Roles I created a role called “lambda-example” and attached the following built-in policy:

AWSLambdaBasicExecutionRole

An additional inline policy was also created and attached to this role. This policy was for DynamoDB read/write permissions. It specifically had these DynamoDB actions attached to it:

BatchGetItem
GetItem
Query
Scan
BatchWriteItem
DeleteItem
PutItem
UpdateItem

With the Role created I am now ready to create the Lambda Function. I create a new function and set it to use NodeJS. I select the Role I created above to the function’s Execution Role. Once the function is created, I paste in the Javascript code from the habits.lambda.js file from the code repository. Note that the API Gateway is going to pass through all HTTP traffic into this Lambda Function. Therefore I need to first be able to distinguish the HTTP Method calls. You can see this in that code as below:

'use strict';
const ddb = new AWS.DynamoDB.DocumentClient();

exports.handler = function (event, context, callback) {
    var response = {
        statusCode: 200,
        headers: {
            "Access-Control-Allow-Origin": "*"
        },
        body: null
    };
    if (event.httpMethod !== null && event.httpMethod !== undefined) {
        if (String(event.httpMethod) == 'POST') {
...
        } else if (String(event.httpMethod) == 'GET') {
            getSummaryAll().then((records) => {
                var habits = records.Items;
                habits.sort(function (a, b) { return b.Date - a.Date; });
                records.Items = habits;
                response.body = JSON.stringify(records);
                callback(null, response);
            }).catch((error) => {
                response.body = "ERROR: Unable to get records from database. " + error.message;
                callback(null, response);
            });
        } else {
            response.body = "ERROR: Unsupported HTTP Method of " + event.httpMethod;
            response.statusCode = 405;
            callback(null, response);
        }
...

Note that I am building up a response object that includes the CORS headers. Although API Gateway automatically enables CORS for some HTTP requests, it is not applied to all (such as GET requests). Therefore I include it here for redundancy.

In this example I only handle the HTTP POST and GET requests. In the GET request section I am retrieving data from DynamoDB using a promise function. This is shown below. Note that due to the way I setup the DynamoDB table keys I had to re-sort the results in the function before sending the results back to the client.

function getSummaryAll() {
    return ddb.scan({
        TableName: 'HabitsSummary'
    }).promise();
}
...
function addHabit(action, comments, date) {
    var pst = dateNow();
    return ddb.put({
        TableName: 'Habits',
        Item: {
            DateTime: pst.toISOString(),
            Date: parseInt(date),
            Action: action,
            Comments: comments
        },
    }).promise();
}

In the above sample code you can also see how a record is added to DynamoDB using the “put” function. Items are sent to DynamoDB as JSON.

With the Lambda Function defined we are now ready to create the API Gateway endpoint. I create a new API and then create a new Resource. This is the actual endpoint for the API. I am going to use a single endpoint for this project that handles all HTTP requests. Therefore I configure it as “proxy resource” and also enable it to allow CORS.

With the new endpoint resource I select it’s Method Execution to be a Lambda Function and point it to my function created in the above section. This is all that is needed for the API Gateway and now we’re ready to setup the DynamoDB database.

Setting up DynamoDB

For this project we have two tables that are created in DynamoDB. The main table “Habits” has a single partition key called DateTime where I will store the UTC timestamps. The other fields that are added are defined in the Lambda Function so in the DynamoDB console I only need to define the single DateTime field. The second table is “HabitsSummary” and this table is needed because out-of-the-box DynamoDB (and NoSQL for that matter) does not support aggregation methods such as ‘count’ or ‘group-by’ like traditional relational databases. So instead, I’m creating what is like a Materialized View and storing the counts in the separate summary table. It only has a single partition key called “Date” which is also a field in the “Habits” table.

Habits

DateTime
Date
Action
Comments

HabitsSummary

Date
Status
Attempts (count field)

Frontend HTML using jQuery

The final part of this project is creating the frontend web page. Its a very simple page using Bootstrap and jQuery. It does an AJAX call into the API Gateway to get the table list and post the updates. Some of the code is shown below but refer to the github repository for the full example.

...
      $.getJSON(url, function (data) {
        var success = 0;
        var fail = 0;
        var attempt = 0;
        var days = 0;

        $.each(data.Items, function (i, item) {
          var attempts = parseInt(item.Attempts);
          switch (item.Status) {
            case "success":
              success++;
              break;
            case "fail":
              fail++;
              break;
          }
          if (days < 30) { attempt = attempt + attempts; }
          days++;

          var row = `${item.Date}${drawStatus(item.Status)}${drawFire(attempts)}

`; $(“#tableBody”).append(row); }); var percentSuccess = success / days; var percentFail = fail / days; $(“#totalSuccess”).append(`${success}`); $(“#totalFail”).append(`${fail}`); $(“#totalAttempt”).append(`${attempt}`); $(“#totalDays”).append(`<strong”>${days}`); }); … function submitAction(action) { var content = { action: action, comments: $(“#formComments”).val(), date: $(“#formDate”).val() }; $.ajax({ url: url, dataType: “json”, contentType: “application/json;charset=utf-8”, type: “POST”, data: JSON.stringify(content), success: function (result) { success(result); }, error: function (status) { if (status && status.status && status.status == 200) { success(status); } else { alert(“ERROR – unable to submit request. ” + JSON.stringify(status)); console.log(JSON.stringify(status)); } } }); } …

References

Another sample project doing Serverless Application
http://solidfish.com/serverless-app-using-aws-api-dynamodb-lambda-s3-visual-studio-dot-net/

Stackoverflow post on ways to do aggregation with DynamoDB
https://stackoverflow.com/questions/36866902/how-to-use-aggregate-functions-in-amazon-dynamodb

Not part of this project but another option of doing aggregation on top of DynamoDB is to use the AWS EMR service
https://docs.aws.amazon.com/emr/latest/ReleaseGuide/EMRforDynamoDB.html

References

Related